Comparing memory capacity across stimuli requires maximally dissimilar foils: Using deep convolutional neural networks to understand visual working memory capacity for real-world objects.

The capacity of visual working and visual long-term memory plays a critical role in theories of cognitive architecture and the relationship between memory and other cognitive systems. Here, we argue that before asking the question of how capacity varies across different stimuli or what the upper bound of capacity is for a given memory system, it is necessary to establish a methodology that allows a fair comparison between distinct stimulus sets and conditions. One of the most important factors determining performance in a memory task is target/foil dissimilarity. We argue that only by maximizing the dissimilarity of the target and foil in each stimulus set can we provide a fair basis for memory comparisons between stimuli. In the current work we focus on a way to pick such foils objectively for complex, meaningful real-world objects by using deep convolutional neural networks, and we validate this using both memory tests and similarity metrics. Using this method, we then provide evidence that there is a greater capacity for real-world objects relative to simple colors in visual working memory; critically, we also show that this difference can be reduced or eliminated when non-comparable foils are used, potentially explaining why previous work has not always found such a difference. Our study thus demonstrates that working memory capacity depends on the type of information that is remembered and that assessing capacity depends critically on foil dissimilarity, especially when comparing memory performance and other cognitive systems across different stimulus sets.

Sequential encoding aids working memory for meaningful objects' identities but not for their colors.

Previous studies have found that real-world objects' identities are better remembered than simple features like colored circles, and this effect is particularly pronounced when these stimuli are encoded one by one in a serial, item-based way. Recent work has also demonstrated that memory for simple features like color is improved if these colors are part of real-world objects, suggesting that meaningful objects can serve as a robust memory scaffold for their associated low-level features. However, it is unclear whether the improved color memory that arises from the colors appearing on real-world objects is affected by encoding format, in particular whether items are encoded sequentially or simultaneously. We test this using randomly colored silhouettes of recognizable versus unrecognizable scrambled objects that offer a uniquely controlled set of stimuli to test color working memory of meaningful versus non-meaningful objects. Participants were presented with four stimuli (silhouettes of objects or scrambled shapes) simultaneously or sequentially. After a short delay, they reported either which colors or which shapes they saw in a two-alternative forced-choice task. We replicated previous findings that meaningful stimuli boost working memory performance for colors (Exp. 1). We found that when participants remembered the colors (Exp. 2) there was no difference in performance across the two encoding formats. However, when participants remembered the shapes and thus identity of the objects (Exp. 3), sequential presentation resulted in better performance than simultaneous presentation. Overall, these results show that different encoding formats can flexibly impact visual working memory depending on what the memory-relevant feature is.

Local but not global graph theoretic measures of semantic networks generalize across tasks.

"Dogs" are connected to "cats" in our minds, and "backyard" to "outdoors." Does the structure of this semantic knowledge differ across people? Network-based approaches are a popular representational scheme for thinking about how relations between different concepts are organized. Recent research uses graph theoretic analyses to examine individual differences in semantic networks for simple concepts and how they relate to other higher-level cognitive processes, such as creativity. However, it remains ambiguous whether individual differences captured via network analyses reflect true differences in measures of the structure of semantic knowledge, or differences in how people strategically approach semantic relatedness tasks. To test this, we examine the reliability of local and global metrics of semantic networks for simple concepts across different semantic relatedness tasks. In four experiments, we find that both weighted and unweighted graph theoretic representations reliably capture individual differences in local measures of semantic networks (e.g., how related pot is to pan versus lion). In contrast, we find that metrics of global structural properties of semantic networks, such as the average clustering coefficient and shortest path length, are less robust across tasks and may not provide reliable individual difference measures of how people represent simple concepts. We discuss the implications of these results and offer recommendations for researchers who seek to apply graph theoretic analyses in the study of individual differences in semantic memory.

Probabilistic and rich individual working memories revealed by a betting game.

When asked to remember a color, do people remember a point estimate (e.g., a particular shade of red), a point estimate plus an uncertainty estimate, or are memory representations rich probabilistic distributions over feature space? We asked participants to report the color of a circle held in working memory. Rather than collecting a single report per trial, we had participants place multiple bets to create trialwise uncertainty distributions. Bet dispersion correlated with performance, indicating that internal uncertainty guided bet placement. While the first bet was on average the most precisely placed, the later bets systematically shifted the distribution closer to the target, resulting in asymmetrical distributions about the first bet. This resulted in memory performance improvements when averaging across bets, and overall suggests that memory representations contain more information than can be conveyed by a single response. The later bets contained target information even when the first response would generally be classified as a guess or report of an incorrect item, suggesting that such failures are not all-or-none. This paradigm provides multiple pieces of evidence that memory representations are rich and probabilistic. Crucially, standard discrete response paradigms underestimate the amount of information in memory representations.

Perceptual Awareness Occurs Along a Graded Continuum: No Evidence of All-or-None Failures in Continuous Reproduction Tasks.

Does sensory information reach conscious awareness in a discrete, all-or-nothing manner or a gradual, continuous manner? To answer this question, we examined behavioral performance across four different paradigms that manipulate visual awareness: the attentional blink, backward masking, the Sperling iconic memory paradigm, and retro-cuing. We then asked how well we could account for participants' (N = 112 adults) behavior using a signal detection framework that factors in psychophysical scaling to model participants' responses along a single continuum. We found that this model easily accounted for the data from each of these diverse paradigms. Moreover, we reanalyzed the data from prior studies that had posited a discrete view of perceptual awareness and found that our continuous signal detection model outperformed the models that had been used to support an all-or-nothing view of consciousness. This set of data is consistent with the idea that conscious awareness occurs along a graded continuum.

A quantitative model of ensemble perception as summed activation in feature space.

Ensemble perception is a process by which we summarize complex scenes. Despite the importance of ensemble perception to everyday cognition, there are few computational models that provide a formal account of this process. Here we develop and test a model in which ensemble representations reflect the global sum of activation signals across all individual items. We leverage this set of minimal assumptions to formally connect a model of memory for individual items to ensembles. We compare our ensemble model against a set of alternative models in five experiments. Our approach uses performance on a visual memory task for individual items to generate zero-free-parameter predictions of interindividual and intraindividual differences in performance on an ensemble continuous-report task. Our top-down modelling approach formally unifies models of memory for individual items and ensembles and opens a venue for building and comparing models of distinct memory processes and representations.

There Is no Theory-Free Measure of "Swaps" in Visual Working Memory Experiments.

Visual working memory is highly limited, and its capacity is tied to many indices of cognitive function. For this reason, there is much interest in understanding its architecture and the sources of its limited capacity. As part of this research effort, researchers often attempt to decompose visual working memory errors into different kinds of errors, with different origins. One of the most common kinds of memory error is referred to as a "swap," where people report a value that closely resembles an item that was not probed (e.g., an incorrect, non-target item). This is typically assumed to reflect confusions, like location binding errors, which result in the wrong item being reported. Capturing swap rates reliably and validly is of great importance because it permits researchers to accurately decompose different sources of memory errors and elucidate the processes that give rise to them. Here, we ask whether different visual working memory models yield robust and consistent estimates of swap rates. This is a major gap in the literature because in both empirical and modeling work, researchers measure swaps without motivating their choice of swap model. Therefore, we use extensive parameter recovery simulations with three mainstream swap models to demonstrate how the choice of measurement model can result in very large differences in estimated swap rates. We find that these choices can have major implications for how swap rates are estimated to change across conditions. In particular, each of the three models we consider can lead to differential quantitative and qualitative interpretations of the data. Our work serves as a cautionary note to researchers as well as a guide for model-based measurement of visual working memory processes.

No Fixed Limit for Storing Simple Visual Features: Realistic Objects Provide an Efficient Scaffold for Holding Features in Mind.

Prominent theories of visual working memory postulate that the capacity to maintain a particular visual feature is fixed. In contrast to these theories, recent studies have demonstrated that meaningful objects are better remembered than simple, nonmeaningful stimuli. Here, we tested whether this is solely because meaningful stimuli can recruit additional features-and thus more storage capacity-or whether simple visual features that are not themselves meaningful can also benefit from being part of a meaningful object. Across five experiments (30 young adults each), we demonstrated that visual working memory capacity for color is greater when colors are part of recognizable real-world objects compared with unrecognizable objects. Our results indicate that meaningful stimuli provide a potent scaffold to help maintain simple visual feature information, possibly because they effectively increase the objects' distinctiveness from each other and reduce interference.

Visual hindsight bias for abnormal mammograms in radiologists.

Purpose: Hindsight bias-where people falsely believe they can accurately predict something once they know about it-is a pervasive decision-making phenomenon, including in the interpretation of radiological images. Evidence suggests it is not only a decision-making phenomenon but also a visual perception one, where prior information about an image enhances our visual perception of the contents of that image. The current experiment investigates to what extent expert radiologists perceive mammograms with visual abnormalities differently when they know what the abnormality is (a visual hindsight bias), above and beyond being biased at a decision level. Approach: N=40 experienced mammography readers were presented with a series of unilateral abnormal mammograms. After each case, they were asked to rate their confidence on a 6-point scale that ranged from confident mass to confident calcification. We used the random image structure evolution method, where the images repeated in an unpredictable order and with varied noise, to ensure any biases were visual, not cognitive. Results: Radiologists who first saw an original image with no noise were more accurate in the max noise level condition [area under the curve (AUC)=0.60] than those who first saw the degraded images (AUC=0.55; difference: p=0.005), suggesting that radiologists' visual perception of medical images is enhanced by prior visual experience with the abnormality. Conclusions: Overall, these results provide evidence that expert radiologists experience not only decision level but also visual hindsight bias, and have potential implications for negligence lawsuits.

The role of motion in visual working memory for dynamic stimuli: More lagged but more precise representations of moving objects.

While most visual working memory studies use static stimuli with unchanging features, objects in the real world are often dynamic, introducing significant differences in the surface feature information hitting the retina from the same object over time (e.g., changes in orientation, lighting, shadows). Previous research on dynamic stimuli has shown that change detection is improved if objects obey rules of physical motion, but it is unclear how memory for visual features interacts with object motion. In the current study, we investigated whether object motion facilitates greater temporal integration of continuously changing surface feature information. In a series of experiments, participants were asked to report the final color of continuously changing colored dots that were either moving or stationary on the screen. We found that the reported colors "lagged behind" the physical states of the dots when they were in motion. We also observed that the precision of memory responses was significantly higher for stimuli in the moving condition compared to the stationary condition. Together, these findings suggest that memory representation is improved - but lagged - for moving objects, consistent with the idea that object motion may facilitate integration of object information over longer intervals.

Structured visuospatial representations revealed through serial reproduction.

Working memory is a reconstructive process that requires integrating multiple hierarchical representations of objects. This hierarchical reconstruction allows us to overcome perceptual uncertainty and limited cognitive capacity but yields systematic biases in working memory as individual items are influenced by the ensemble statistics of the scene, or of their particular group. Given the importance of the hierarchical encoding of a display, we aim to characterize what structures people use to encode visual scenes using a nonparametric data-driven approach. In Experiment 1, we examine visuospatial memory for locations by asking participants to recall the locations of objects in a serial reproduction task. We show that people report items in a more compact structure than they initially were and organize them into clustered spatial groups. In Experiment 2, we explicitly introduce discrete color groups, allowing us to test whether the color feature governs the spatial grouping. We find that the spatial structures were color-contingent. By analyzing color groups, we circumvent the grouping uncertainty in Experiment 1 and further reveal that people compress color groups into collinear structures with similar orientations and equidistant spacing. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Measuring memory is harder than you think: How to avoid problematic measurement practices in memory research.

We argue that critical areas of memory research rely on problematic measurement practices and provide concrete suggestions to improve the situation. In particular, we highlight the prevalence of memory studies that use tasks (like the "old/new" task: "have you seen this item before? yes/no") where quantifying performance is deeply dependent on counterfactual reasoning that depends on the (unknowable) distribution of underlying memory signals. As a result of this difficulty, different literatures in memory research (e.g., visual working memory, eyewitness identification, picture memory, etc.) have settled on a variety of fundamentally different metrics to get performance measures from such tasks (e.g., A', corrected hit rate, percent correct, d', diagnosticity ratios, K values, etc.), even though these metrics make different, contradictory assumptions about the distribution of latent memory signals, and even though all of their assumptions are frequently incorrect. We suggest that in order for the psychology and neuroscience of memory to become a more cumulative, theory-driven science, more attention must be given to measurement issues. We make a concrete suggestion: The default memory task for those simply interested in performance should change from old/new ("did you see this item'?") to two-alternative forced-choice ("which of these two items did you see?"). In situations where old/new variants are preferred (e.g., eyewitness identification; theoretical investigations of the nature of memory signals), receiver operating characteristic (ROC) analysis should be performed rather than a binary old/new task.

You cannot "count" how many items people remember in visual working memory: The importance of signal detection-based measures for understanding change detection performance.

Change detection tasks are commonly used to measure and understand the nature of visual working memory capacity. Across three experiments, we examine whether the nature of the memory signals used to perform change detection are continuous or all-or-none and consider the implications for proper measurement of performance. In Experiment 1, we find evidence from confidence reports that visual working memory is continuous in strength, with strong support for an equal variance signal detection model with no guesses or lapses. Experiments 2 and 3 test an implication of this, which is that K should confound response criteria and memory. We found K values increased by roughly 30% when criteria are shifted despite no change in the underlying memory signals. Overall, our data call into question a large body of work using threshold measures, like K, to analyze change detection data. This metric confounds response bias with memory performance and is inconsistent with the vast majority of visual working memory models, which propose variations in precision or strength are present in working memory. Instead, our data indicate an equal variance signal detection model (and thus, d')-without need for lapses or guesses-is sufficient to explain change detection performance. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Amplification in the evaluation of multiple emotional expressions over time.

Social interactions are dynamic and unfold over time. To make sense of social interactions, people must aggregate sequential information into summary, global evaluations. But how do people do this? Here, to address this question, we conducted nine studies (N = 1,583) using a diverse set of stimuli. Our focus was a central aspect of social interaction-namely, the evaluation of others' emotional responses. The results suggest that when aggregating sequences of images and videos expressing varying degrees of emotion, perceivers overestimate the sequence's average emotional intensity. This tendency for overestimation is driven by stronger memory of more emotional expressions. A computational model supports this account and shows that amplification cannot be explained only by nonlinear perception of individual exemplars. Our results demonstrate an amplification effect in the perception of sequential emotional information, which may have implications for the many types of social interactions that involve repeated emotion estimation.

An adaptive perspective on visual working memory distortions.

When holding multiple items in visual working memory, representations of individual items are often attracted to, or repelled from, each other. While this is empirically well-established, existing frameworks do not account for both types of distortions, which appear to be in opposition. Here, we demonstrate that both types of memory distortion may confer functional benefits under different circumstances. When there are many items to remember and subjects are near their capacity to accurately remember each item individually, memories for each item become more similar (attraction). However, when remembering smaller sets of highly similar but discernible items, memory for each item becomes more distinct (repulsion), possibly to support better discrimination. Importantly, this repulsion grows stronger with longer delays, suggesting that it dynamically evolves in memory and is not just a differentiation process that occurs during encoding. Furthermore, both attraction and repulsion occur even in tasks designed to mitigate response bias concerns, suggesting they are genuine changes in memory representations. Together, these results are in line with the theory that attraction biases act to stabilize memory signals by capitalizing on information about an entire group of items, whereas repulsion biases reflect a tradeoff between maintaining accurate but distinct representations. Both biases suggest that human memory systems may sacrifice veridical representations in favor of representations that better support specific behavioral goals. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Guidance of attention by working memory is a matter of representational fidelity.

Items that are held in visual working memory can guide attention toward matching features in the environment. Predominant theories propose that to guide attention, a memory item must be internally prioritized and given a special template status, which builds on the assumption that there are qualitatively distinct states in working memory. Here, we propose that no distinct states in working memory are necessary to explain why some items guide attention and others do not. Instead, we propose variations in attentional guidance arise because individual memories naturally vary in their representational fidelity, and only highly accurate memories automatically guide attention. Across a series of experiments and a simulation we show that (a) items in working memory vary naturally in representational fidelity; (b) attention is guided by all well-represented items, though frequently only one item is represented well enough to guide; and (c) no special working memory state for prioritized items is necessary to explain guidance. These findings challenge current models of attentional guidance and working memory and instead support a simpler account for how working memory and attention interact: Only the representational fidelity of memories, which varies naturally between items, determines whether and how strongly a memory representation guides attention. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Rethinking the Ranks of Visual Channels.

Data can be visually represented using visual channels like position, length or luminance. An existing ranking of these visual channels is based on how accurately participants could report the ratio between two depicted values. There is an assumption that this ranking should hold for different tasks and for different numbers of marks. However, there is surprisingly little existing work that tests this assumption, especially given that visually computing ratios is relatively unimportant in real-world visualizations, compared to seeing, remembering, and comparing trends and motifs, across displays that almost universally depict more than two values. To simulate the information extracted from a glance at a visualization, we instead asked participants to immediately reproduce a set of values from memory after they were shown the visualization. These values could be shown in a bar graph (position (bar)), line graph (position (line)), heat map (luminance), bubble chart (area), misaligned bar graph (length), or 'wind map' (angle). With a Bayesian multilevel modeling approach, we show how the rank positions of visual channels shift across different numbers of marks (2, 4 or 8) and for bias, precision, and error measures. The ranking did not hold, even for reproductions of only 2 marks, and the new probabilistic ranking was highly inconsistent for reproductions of different numbers of marks. Other factors besides channel choice had an order of magnitude more influence on performance, such as the number of values in the series (e.g., more marks led to larger errors), or the value of each mark (e.g., small values were systematically overestimated). Every visual channel was worse for displays with 8 marks than 4, consistent with established limits on visual memory. These results point to the need for a body of empirical studies that move beyond two-value ratio judgments as a baseline for reliably ranking the quality of a visual channel, including testing new tasks (detection of trends or motifs), timescales (immediate computation, or later comparison), and the number of values (from a handful, to thousands).

The role of meaning in visual working memory: Real-world objects, but not simple features, benefit from deeper processing.

Visual working memory is a capacity-limited cognitive system used to actively store and manipulate visual information. Visual working memory capacity is not fixed, but varies by stimulus type: Stimuli that are more meaningful are better remembered. In the current work, we investigate what conditions lead to the strongest benefits for meaningful stimuli. We propose that in some situations participants may try to encode the entire display holistically (i.e., in a quick "snapshot"). This may lead them to treat objects as simply meaningless, colored "blobs", rather than individually and in a high-level way, which could reduce benefits of meaningful stimuli. In a series of experiments, we directly test whether real-world objects, colors, perceptually matched less-meaningful objects, and fully scrambled objects benefit from deeper processing. We systematically vary the presentation format of stimuli at encoding to be either simultaneous-encouraging a parallel, "take-a-quick-snapshot" strategy-or present the stimuli sequentially, promoting a serial, each-item-at-once strategy. We find large advantages for meaningful objects in all conditions, but find that real-world objects-and to a lesser degree lightly scrambled, still meaningful versions of the objects-benefit from the sequential encoding and thus deeper, focused-on-individual-items processing, while colors do not. Our results suggest single-feature objects may be an outlier in their affordance of parallel, quick processing, and that in more realistic memory situations, visual working memory likely relies upon representations resulting from in-depth processing of objects (e.g., in higher-level visual areas) rather than solely being represented in terms of their low-level features. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Greater Visual Working Memory Capacity for Visually Matched Stimuli When They Are Perceived as Meaningful.

Almost all models of visual working memory-the cognitive system that holds visual information in an active state-assume it has a fixed capacity: Some models propose a limit of three to four objects, where others propose there is a fixed pool of resources for each basic visual feature. Recent findings, however, suggest that memory performance is improved for real-world objects. What supports these increases in capacity? Here, we test whether the meaningfulness of a stimulus alone influences working memory capacity while controlling for visual complexity and directly assessing the active component of working memory using EEG. Participants remembered ambiguous stimuli that could either be perceived as a face or as meaningless shapes. Participants had higher performance and increased neural delay activity when the memory display consisted of more meaningful stimuli. Critically, by asking participants whether they perceived the stimuli as a face or not, we also show that these increases in visual working memory capacity and recruitment of additional neural resources are because of the subjective perception of the stimulus and thus cannot be driven by physical properties of the stimulus. Broadly, this suggests that the capacity for active storage in visual working memory is not fixed but that more meaningful stimuli recruit additional working memory resources, allowing them to be better remembered.